Repository navigation
Use a cloud based model - #1096
Conversation
The responses API is more recent has support for more features such as tool use, however it isn't supported as widely by cloud providers. Specifically, the qwen models we're considering that is hosted by Cloudflare only supports the chat API. I think the chat API is sufficient for our use case. This also makes the mocking easier in the tests with a helper function. Responses vs chat API: https://developers.openai.com/api/docs/guides/migrate-to-responses Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Test coverage94.18% line coverage reported by SimpleCov. |
ApprovabilityVerdict: Needs Human Approval Macroscope's review found this PR requires human approval — This switches learner error explanations to a Cloudflare-hosted model, sending code and traces through its gateway. Human confirmation is needed that the rollout scope and provider/gateway logging and retention controls are appropriate for learner data before production activation. Needs human approval because:
You can add or adjust custom eligibility rules. Learn more. |
This qwen model is a suggestion as it's one of the closest to the ones we have previously tested and has a reasonable cost. We will run the evaluator against it to confirm it's suitable. Note I took out the reasoning config as it changes the structure of the response and causes errors (I'm not sure why). I'll add this back in a future commit
I thought the other method was getting quite complex
When testing we found that reasoning increased cost and time while providing little benefit for the kinds of data we expect. Initially I tried disabling this using the 'reasoning' parameter but this changes the structure of the response. I've not been able to work out why, but assume it is the way this particular model is hosted or configured. Instead I've switched to an alternative way to disable reasoning using the /no_think command.
Having a low temperature makes the models relatively more predictable which is preferred for our needs. I tested this by asking the model to name an animal. I still saw variance in responses with a low temperature, but I saw less than with a higher temperature so I assume it's working.
In english, it's common to estimate each token as approximately 4 characters long so should be set to about 1/4 of the max result length. I've added a generous buffer on top of this.
d50aaf7 to
76b958a
Compare
| { role: 'user', content: JSON.dump({ message:, trace:, runtime:, code: }) } | ||
| ], | ||
| temperature: 0, | ||
| max_completion_tokens: MAX_COMPLETION_TOKENS, |
There was a problem hiding this comment.
I wonder if we need this and the MAX_RESULT_LENGTH_CHARS check?
There was a problem hiding this comment.
Yes, I think you shouldn't as the number of output tokens will be correlated with output length. I can remove
There was a problem hiding this comment.
I've removed the MAX_RESULT_LENGTH_CHARS check.
|
Originally this was using qwen3-30b-a3b-fp8, cloudlfare are removing this model this month so I've switched this to using qwen3.8-27b. There's a couple of changes I need to make to get this working - I think there's a different way to disable reasoning and the prompts have to be combined. |
Originally I was testing with an older qwen model. I've updated to 3.8 and there are two differences with the API: + Only a single system prompt is allowed + Disabling thinking/reasoning is done in a different way
The token length check seems like a reliable way to limit the size of responses so I've removed the character check
Status
What's changed?
I've adapted what we've already done to use a Cloudflare hosted model instead of a local one
See commits for more
Steps to perform after deploying to production
BYPASS_FEM_INFERENCE